Audio messaging interface on messaging platform

ABSTRACT

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for audio messaging interface for messaging platform. One of the methods includes receiving, by a first client on a first user device, a request to record an audio message, wherein the first client is configured to provide a user interface for the platform for a user using the first user device who is logged in to a user account on the platform; recording audio through a microphone of the first user device; generating a platform message by (i) generating a video file that includes the recorded audio as an audio portion of the video file and programmatically generated minimal video content as a video portion of the video file, and (ii) including the video file in the platform message; and posting, by the first client, the platform message to the platform, in response to a post request.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a divisional of, and claims priority to, U.S. patentapplication Ser. No. 16/904,526, filed on Jun. 17, 2020, and which isincorporated herein by reference.

BACKGROUND

This specification relates to generating and displaying audiovisualcontent on online social messaging platforms. Social messaging platformsand network-connected personal computing devices allow users to createand share content across multiple devices in real-time.

Sophisticated mobile computing devices such as smartphones and tabletsmake it easy and convenient for people, companies, and other entities touse online social networking messaging platforms and applications.Popular social messaging platforms generally provide functionality forusers to draft and post messages, both synchronously and asynchronously,to other users. Other common features include the ability to postmessages that are visible to one or more identified other users of theplatform, or even publicly to any user of the platform, without specificdesignation by the authoring user.

SUMMARY

This specification describes technologies for a user interface thatfacilitates recording, editing, and posting audiovisual content to anonline social messaging platform. The technologies generally involvereceiving a request by a user device to record audio for an audiomessage. An audio message is a message of an online social messagingplatform that includes audio content, including videos that have asoundtrack. In response to the request, the user device records audio bya microphone on the user device. After the recording is finished, theuser device automatically generates an audio message with the recordedaudio, and posts the audio message to the platform.

A user device can play back the recorded audio along with a graphic oranimation that was displayed while the audio was recording. The userdevice can receive a confirmation that the audio recording is ready tobe posted as part of an audio message. The user device can record theaudio and generate a corresponding video. The video includes asoundtrack of the audio recording and video frames representing theanimation played while the user device recorded the audio. In someimplementations, the user device receives inputs indicating edits tomake to the video. The user device, in response to the inputs, canmodify audiovisual properties of the video, e.g., by adding soundeffects to the video's soundtrack. The video frames can be of minimalvideo content, such as a low resolution image or solid color, acting asa placeholder for encoding the recorded audio in an audiovisual—insteadof purely audio—format. Codecs for audiovisual file formats, e.g.,MPEG-4, are universally available on a variety of different devices, andthe user device leverages this available over purely audio formats toencode the recorded audio.

In this way, network bandwidth utilization is reduced, because the audiomessage can be transmitted having the minimal video content (e.g.,having kilobytes of data) to the receiving user device, and thedetermination and rendering of content to replace the video content canbe performed client-side by the user device. User experience is improvedwhile keeping network utilization down, all while still enabling therecorded audio to be processed and played back using any audiovisualcodec.

The user device, as part of generating the video with the audiorecording soundtrack, can process the audio recording using suitablespeech-to-text technology to generate text corresponding to speechrecorded in the audio. As part of posting the audio message to theplatform, the user device can further include at least a portion of thegenerated text in the audio message, along with the generated video.

The user device can record the audio, display the corresponding graphicor animation, and then send the audio recording to one or more serversimplementing the platform. The server(s) can generate a video for theaudio recording, using a locally stored or rendered graphic or animationcorresponding to the graphic or animation displayed on the user device.The generated video is generated by the platform server(s) using thelocally stored or rendered graphic or animation, which can have a pixelresolution, e.g., 3840×2160 (“4K” resolution), that is higher than thepixel resolution of the user device. The platform server(s) thenautomatically post the audio message with the server-generated videowhich can be of a higher resolution than the resolution of a userdevice-generated video.

In some implementations, a user device receives an audio message,wherein the recorded audio is encoded as a video with frames havingminimal video content. When the user device plays back the audiomessage, the user device discards the video frames and instead renders agraphic, image, or animation to accompany the recorded audio of themessage as it is played back. By rendering a graphic, image, oranimation in time for playback, the user device obviates the need fordetailed, high-resolution video content from being stored as part of theaudio message, and does so without sacrificing content quality in theuser experience.

The subject matter described in this specification can be implemented inparticular embodiments so as to realize one or more of the followingadvantages. Audio can be efficiently recorded and edited before beingposted to a social messaging platform. A user interface as described inthis specification facilitates recording sounds originating near arecording user device. Audio messages facilitate disseminatinginformation to the platform by including audio which can provide moreinformation and context than messages that include text alone.

After recording, the audio can be automatically played back and the userinterface provides several options for receiving user input formodifying the recorded audio at different points in time of therecording. After the audio is recorded and modified, the recording canbe automatically posted to a messaging platform as a message in amessage stream, accessible to users of the platform. The user device canautomatically be prepared to record another audio message by the sameuser interface, which can improve the rate at which information isdisseminated to the platform because of the stream-lined and efficientuser experience for generating and editing the messages.

Audio recorded can also be converted to text and automatically posted tothe messaging platform as a message containing the text and accompanyingaudio. In this way, information dissemination is further improved,because audio messages contain information in multiple formats, ready tobe consumed in different ways by users of the platform.

The user device can display the graphic or animation at a pixelresolution that is the same as, or less than, the native resolution ofthe user device's display. However, by sending the audio recording toone or more servers implementing the platform to generate thecorresponding video, the server or servers can generate the video at aresolution that is potentially higher than what the user device alonecould generate. Either on one or more servers or on the authoringdevice, the recording can be rendered into multiple versions, multipleformats, form factors, or resolutions that are optimized for specificdevices to consume. The optimization can be based on, for example, filesize, speed in downloading or uploading the recording, playback qualityof the recording on the user device, or a combination of these or othercriteria.

The details of one or more embodiments of the subject matter of thisspecification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages of thesubject matter will become apparent from the description, the drawings,and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example online social messaging platform andexample user devices.

FIG. 2 is a flowchart of an example process for generating an audiomessage for an online social messaging platform.

FIG. 3 illustrates an example user interface for recording audio andposting audio messages to the platform.

FIG. 4 illustrates an example user interface for modifying an audiorecording before posting the audio recording as part of an audio messageto the platform.

FIG. 5 illustrates an example user interface displaying a message streamthat includes an audio message.

FIG. 6 is a flowchart of an example process for generating text fromrecorded audio and posting a message that includes the generated text toan online social messaging platform.

Like reference numbers and designations in the various drawings indicatelike elements.

DETAILED DESCRIPTION

FIG. 1 illustrates an example online social messaging system includingan online social messaging platform 100 and user devices 104 a-104 noperated by users 102 a-102 n.

A user may be an account holder of a user account, or an authorized userof an account, on the platform 100. A user device can be anyInternet-connected device, e.g., a laptop or desktop computer, asmartphone, or an electronic tablet. The user device can be connected tothe Internet through a mobile network, through an Internet serviceprovider (ISP), or otherwise. A user device includes a microphone, adisplay, and client software to record audio and video.

User device 104 a is configured with client software 106 a, whichincludes an audio message engine 108 a. The functionality of the audiomessage engine 108 a is described in detail, below. The client software106 a also implements a user interface for the user device 104 a toreceive input from the user 102 a, and to generate and modify audiomessages before posting the messages to the platform 100. In someimplementations and as described in more detail below, the clientsoftware 106 a is configured to generate and modify messages containingtext corresponding to speech recorded by the user device 104 a, and alsoto post the messages to the platform 100. The client software 106 a isalso configured to provide a user interface for the user 102 a tointeract with the platform 100.

The client software may be a web browser or an HTML (hypertext markuplanguage) document rendered by a web browser. Or the client software maybe a script or another type of software code, e.g., JavaScript code orJava code. Or the client software may be dedicated software, e.g., aninstalled app or installed application that is designed to workspecifically with the platform 100. Or the client software may be orinclude a Short Messaging Service (SMS) interface, an instant messaginginterface, an email-based interface, or an API function-based interface,for example.

The platform 100 is implemented on one or more platform servers 110a-110 m that are each implemented on one or more computers located atone or more locations. Each platform server 110 a-110 m supportsconnections to and from the user devices over wired or wireless networksof one or more data communication networks 120. Platform server 110 a isconfigured with audio message engine 112 a, the functionality of whichis described below.

Platform software and client software 106 a-106 n are configured toenable users 102 a-102 n to use the platform 100 to post messages 122 tothe platform and to use the platform 100 to receive messages 124 postedby other users, as well as to engage in other social activities. In thisspecification, a message posted to the platform 100 can contain datarepresenting content provided by the author of the message. An authorcan be a user of the platform, who is identified by a corresponding useraccount. A message may be a container data type storing the contentdata. The types of data that may be stored in a message include text,graphics, images, video, and computer code, e.g., uniform resourcelocators (URLs), for example.

Also in this specification, an audio message is a message that includesaudio content. The audio can be stored in any audio format, e.g., WAV(Waveform Audio File), FLAC (Free Lossless Audio Codex), or MP3. Audiocontent can also be stored in an audiovisual format, e.g., MP4, AVI(Audio Video Interleave), or WMV (Windows Media Video). Purely audiocontent, i.e., audio content without corresponding video, may still bestored in an audiovisual format. One reason to store audio content in avideo file is because user devices are generally configured to encodeand decode universally accepted audiovisual formats, e.g., MP4, but notother strictly audio formats, e.g., WAV.

When an audio message includes a video file, the video file includes anaudio portion and video content as a video portion making up the videofile. Video content includes one or more video frames. The frames can beof a single graphic or animation, e.g., an animated waveform, ananimated avatar, an animated graphic, that is looped for the duration ofthe audio content.

In some implementations, the video content is minimal video content.Minimal video content refers to video content that serves as aplaceholder when generating the audio message to include a video. Theminimal video content contributes much less to the file size versus theaccompanying audio, but allows audio to be saved in an audiovisualformat to allow for broader accessibility by devices with universallyaccepted codecs, such as MPEG-4. For example, the minimal video contentcan be a solid color or unchanging graphic or image. To further reducethe memory size of the minimal video content, the minimal video contentcan include an unchanging graphic or image that is monochrome, obviatingadditional pixel data required in full-color graphics or images. Inaddition, the minimal video content can be a looped animation, such asan animated avatar or undulating waveform.

The minimal video content can be programmatically generated to providevideo content for encoding the recorded audio using an audiovisualcodec. After generating the audio message and posting the message to theplatform, a receiving user device can receive the audio message as partof a message stream, determine that the audio message includes minimalvideo content, and replace the minimal video content with higherresolution and more dynamic video content for display, which is renderedby the client installed on the receiving user device. To keep the memorysize of the minimal video content small, the minimal video content isnot “natural,” i.e., of video content recorded, e.g., by a camera on theuser device. Instead, the minimal video content is “artificial,” i.e.,programmatically generated, using any one of a plurality of techniques,e.g., generating frames of video content having a single color orpattern, e.g., a waveform. The artificial nature of the minimal videocontent allows the content to be used as a placeholder for video datawhile encoding the recorded audio in an audiovisual format, e.g., mp4,and while keeping the size of the minimal video content small. Videocontent of a video file generated for an audio message can include animage that relates to the audio content in some way, although therelationship is not necessary. The video frames can also includeinformation describing the audio content, e.g., the time the audiocontent was authored by a user, or some user-authored text descriptionof the audio content.

In operation, the client software can access the platform so that a usercan post and receive messages, view, and curate the user's streams, andview and interact with lists of content items. A stream is a stream ofmessages on the platform that meet one or more stream criteria. A streamcan be defined by the stream criteria to include messages posted byusers of the platform.

FIG. 2 is a flowchart of an example process 200 for generating an audiomessage for an online social messaging platform. For convenience, theprocess 200 will be described as being performed by a user deviceprogrammed appropriately in accordance with this specification. Forexample, any one of the user devices 104 a of FIG. 1, appropriatelyprogrammed with the audio message engine 108 a, can perform the process200. In the description below, reference is made to a single userdevice, although it is understood that more than one user device canperform the example process 200 in parallel.

The user device displays 205 a user interface for recording audio andposting audio messages to the platform. Client software of the userdevice is configured to display the user interface. The user interfaceincludes interactable elements, i.e., elements with which a user caninteract, for the user to begin and end an audio recording. FIG. 3,below, illustrates an example user interface on the user device forrecording audio and posting audio messages to the platform.

The user device records 210 audio using a microphone. The user devicereceives input at the user interface to begin recording audio, e.g., byrecording audio while receiving a continuous touch input or mouse pressat a user interface element configured to receive input to start andstop audio recording. In some implementations, the user interface isconfigured to receive input to start and stop audio recording by buttonpress or some other element to toggle audio recording.

The audio content recorded can be any audio within perceptual range ofthe microphone of the user device. For example, the audio content can beof a live physical event occurring proximate to the user device, e.g., aconference, a show, or presentation. As another example, the audiocontent can include speech from one or more sources, e.g., the user ofthe user device.

In some implementations, while the user device records audio, the userdevice displays a graphic or animation indicating that the user deviceis actively recording audio. In some implementations, the user devicecan display a modulating waveform while recording the audio. Thewaveform can be a looped, pre-rendered animation, or in someimplementations can be rendered using a wave function that includesrandom amplitudes and wavelengths to simulate a waveform. In someimplementations, the user device simulates the animated waveform bysampling an amplitude and frequency.

The simulated waveform requires fewer computational resources to producein real-time than a waveform that corresponds to audio being recorded.In some implementations, however, the user device generates the waveformto accurately match the amplitudes and frequencies of the incomingaudio.

Other animations, graphics, and images—both still and animated—arepossible. As another example, the user interface can display an animatedavatar, e.g., of the user or of an animal, while the user device recordsaudio. The animated avatar can be rendered to give the appearance thatthe avatar is speaking or otherwise making a sound, e.g., playing aninstrument.

After recording the audio, the user device generates 215 a video in anyaudiovisual format, e.g., MP4. The soundtrack of the video is therecorded audio, and the frames of the video represent the animation orgraphic displayed on the user device while the user device recorded theaudio.

In some implementations, the frames of the video are of minimal videocontent, e.g., a solid color or still image, as described above withreference to FIG. 1. As described in more detail below, when an audiomessage with a generated video is played back on a requesting userdevice, the client software of the user device can render a graphic,image, or animation to accompany the audio of the message when it isplayed back. The video itself can be of minimal video content, andaccordingly of a smaller file size than if a pre-rendered animation orgraphic was generated as part of the video.

Metadata for the video can include configuration options for modifyinghow the user device renders the video of a requested audio message. Forexample, the metadata can specify the type of graphic or animation torender and display, and options for modifying the appearance, duration,or other visual qualities of the rendered graphic or animation.

Optionally, when the user device finishes recording, the user deviceplays back 220 the video that includes the audio recording. The user hasan opportunity to review the audio as it will be heard when posted aspart of an audio message to the platform. The user interface isconfigured to receive input that causes the user device to post theaudio message. In some implementations, the input that causes the userdevice to post the audio message is also input to stop recording theaudio.

As part of providing the audio recording for playback, the user devicecan render and display an animation or graphic, which may be the same ordifferent from an animation or graphic displayed while the user devicewas recording audio. For example, the user device can render an animatedwaveform that can be scrubbed by user input to play back differentportions of the recording. If an animation or graphic is specified inthe video metadata, then the user device can render and display ananimation or graphic according to the metadata. The user device cansimulate the actual waveform for the audio recording, as describedabove, or the user device can generate an accurate waveform as a visualcounterpart for the audio recording as it is played back for userreview.

In some implementations, the play back of the audio 220 and the videomodification 225 can be repeated multiple times. For example, aftermodifying 225 the video, the video is played back 220 and can be furthermodified 225 and played back again. The alternation of play back andmodification can repeat until the user device receives an input toproceed to post 230 the video to the platform as part of an audiomessage.

In some implementations, instead of playing back the video, the userdevice automatically posts 230 the video to the platform as part of anaudio message. In those implementations, the user device does notreceive input but instead posts the audio message after the video isgenerated. After posting the audio message, the user device can display205 the user interface for recording additional audio. In this way, theuser device can quickly record and post audio messages to the platformwith a simple interface, requiring minimal input from the user.

Optionally, the user device modifies 225 the audio before posting 230 tothe platform. The user device can receive input to modify (i) thesoundtrack of the generated video, (ii) video frames of the video, or(iii) both. The user device continues to receive input until receiving aconfirmatory input, e.g., from a user, indicating that the edit phasehas ended. If the user device receives input for modify the video framesof the video, then the user device can save the edits specified by theinput in the video metadata. Later, when an audio message including thevideo is received by another user device, the user device can render anddisplay an appropriate graphic or animation according to the videometadata.

In general, the user device can modify the generated video according toany audiovisual editing technique. For example, the user device can editaudio properties, e.g., pitch, volume, treble, or bass, of the recordedaudio. As another example, the user device can apply audio filters tothe video, or modify the appearance, e.g., modify color, shape, oranimation style, of the waveform rendered and displayed by a user deviceduring playback.

In some implementations, the user device receives input corresponding todifferent sound effects that can be added to the audio recording. Soundeffects can include short, e.g., 1-4 seconds, recordings of instrumentsor other sources of sound that play in addition to the audio of therecording. Sound effects can be of any predefined length of audio fromother recorded audio.

Sound effects can be inserted or overlaid as part of the audiorecording. The user device receives input specifying a timestamp for theaudio recording during which a sound effect is selected for insertion oroverlaying. When a sound effect is inserted, the audio recording isextended by the length of the sound effect. After the sound effect isplayed, the audio from the recording resumes. When a sound effect isoverlaid over the audio recording, both the sound effect and the audioat the specified timestamp are played at the same time. Multiple soundeffects can be added to the same point in time indicated by thetimestamp.

In some implementations, the user device overlays markers on thewaveform, indicating that a sound effect has been inserted at a certaintime of the recording. The user input for selecting and inserting asound effect can be received by the user device through a drag-and-dropgesture detected by the user device, in which a visual markercorresponding to a particular sound effect is dragged and placed on topof a portion of the waveform also displayed. The visual markers aredisplayed until the user device receives an input from the user tofinish editing. In these implementations, the waveform includes a seriesof points, each corresponding to a timestamp within the duration of theaudio recording.

The user device posts 225 an audio message including the video to theplatform. In some implementations, the user device automatically postsan audio message after finishing the audio recording, without receivingadditional user input. Because editing of the generated video isoptional, in some implementations the user device posts the audiomessage after receiving a post request from the user. A user of the userdevice can provide audio content in real time. As a result, the userdevice facilitates sending audio messages to the platform quickly andusing a simple interface.

Instead of generating the video, in some implementations the user devicerecords audio and sends the audio to the messaging platform. A platformserver, e.g., the platform server 110 a of FIG. 1, is configured toprocess the received audio recording through the audio message engine112 a and to generate a video with the audio recording soundtrack, asdescribed above. By sending the audio recording to the platform, theplatform can generate a higher pixel resolution video than what ispossible by the native resolution of the user device.

After the platform generates the video, the platform can post an audiomessage with the video on behalf of the user of the user device. Whenthe audio message is posted, the included video is posted at a sourceresolution equal to higher than the resolution of the display of anyuser device. When the audio message is retrieved by a user device, themessage is retrieved and down-sampled as necessary to allow the userdevice to display the included video at the device's native resolution,which can be equal to or less than the source resolution used togenerate the video.

In some implementations in which the audio recording is sent to theplatform for generating the video, the user device also sends videometadata specifying edits received by the user device for editing thevideo. The platform can generate the video for the audio messageaccording to the received audio recording and metadata.

A user device receiving the audio message can determine whether thevideo content of the video file of the audio message is ofprogrammatically generated minimal video content. In response, the userdevice can play back the video file in the audio message with videocontent rendered by the user device, in place of the minimal videocontent, and also play back the audio recording. The rendered videocontent can be locally stored on the user device or generated by theuser device, and is generally of a higher resolution than the minimalvideo content. Rather than transmit the audio message with the higherresolution (and therefore, larger) video, the receiving user device cansubstitute the minimal video content with the rendered video contentlocally and before display on the user device, thereby reducing networkutilization.

Generally, the user device determines that the audio message includesminimal video content as described in this specification, by comparingdata specifying the video portion of the video in the audio message witha known signature for data specifying minimal video content. Forexample, minimal video content can be generated and inserted in audiomessages of the platform uniformly, meaning that all the minimal videocontent is the same from message-to-message. In this example, thesignature can be a hash value of the minimal video content, and theclient for the receiving user device can compare the hash with a knownhash value to determine that the audio message includes minimal videocontent.

In some implementations, the minimal video content varies frommessage-to-message, but can include a flag indicating that that thecontent has been programmatically generated, and therefore being used asa placeholder to allow for encoding the recorded audio in an audiovisualformat, as described above. In these implementations, the client for theuser device receiving the audio message determines that the messageincludes minimal video content by checking for the presence or absenceof the designated flag.

FIG. 3 illustrates an example user interface 300 for recording audio andposting audio messages to the platform. This and any other interfacedescribed in this specification can be implemented with any additionalfeature to facilitate user accessibility, e.g., high-contrast displayand speech-to-text functionality for visually-impaired users, closedcaptioning for audio content provided as part of a user interface forhearing-impaired users, and voice input functionality as an alternativeor in addition to tactile input for interacting with elements of thedisplayed user interface. The user interface 300 includes a recordinginput element 305, e.g., a button, in which a user of a user device 310displaying the interface 300 can interact with to begin recording audiousing a microphone 315. A user can interact with the interface 300through tactile input, e.g., by tapping the display of the user device310, or in any other manner, e.g., voice input. As the microphone 315records audio, the user device 310 displays an animated waveform 320.The animated waveform 320 can modulate or pulsate as a visual indicatorthat audio is actively being recorded. In some implementations, the userinterface 300 includes a message posting element for receiving input topost the audio recording as part of an audio message.

In some implementations, the user interface for recording and postingaudio can be condensed as a single UI element that, when interacted withby the user, causes the user device to record audio and post audiorecording as described above. The single UI element can be a button, andin some implementations, the button can be overlaid to “float” over acurrent user interface displayed on the user device. The button can beany shape or size up to the size of the display of the user device. Inthese implementations, the button can be moved about the display of theuser device independently of the underlying user interface.

FIG. 4 illustrates an example user interface 400 for modifying an audiorecording before posting it as part of an audio message to the platform.The user interface 400 includes a waveform 405 and a time slider element410. The waveform 405 and the time slider element 410 span the width ofthe display of the user device 310 and can be scrubbed backwards andforwards in time according to user input, e.g., by a touch-and-draggesture from a user.

Although the waveform 405 is shown as a sinusoidal wave, the user devicecan display the waveform 405 in other suitable wave formats, e.g.,square waves, sawtooth waves, or triangular waves; or in other non-waveformats, such as pulsating bars arranged along a line or circle. As thewaveform 405 is scrubbed, the time slider element 410 is also scrubbedto indicate times of the audio recording corresponding to the visibleportion of the waveform 405.

The user interface 400 also includes sound effect markers 415 that auser interacts with to add, remove, and modify the audio recording. Asound effect marker represents a short, e.g., 1 to 4 seconds, soundeffect. A sound effect marker can represent a corresponding sound effectpictorially, e.g., a picture of an instrument played, or textually,e.g., the name of the instrument played in the sound effect. The userinterface 400 can receive a first input selecting a sound effect marker,and a second input selecting a portion in the waveform 405 where thesound effect should be played in the audio recording.

The waveform 405 is annotated with audio sound effect markers 420 wherethe waveform 405 is modified by one or more sound effects. The width ofeach marker 420 indicates the length of the corresponding sound effectwhen played back. The user interface 400 can receive additional input toadd, remove, or modify the position of the markers 420 after they areplaced on the waveform 405.

The user interface 400 also includes a playback element 425. When theplayback element 425 receives input, the user device 310 can start orstop playback of the audio recording from a current time indicated by atime marker 430. The user interface 400 also includes a post element 425that can receive input and cause the user device to post an audiomessage including the audio recording—with any edits—to the platform.

FIG. 5 illustrates an example user interface 500 displaying a messagestream 505 that includes an audio message 510. The audio message 510plays its corresponding audio content when the message 510 is in view onthe display of the user device 310. Alternatively, the audio message 510does not play its audio content unless the user interface 500 receivesan input, e.g., a gesture to the user interface 500 indicating that theaudio message 510 has been selected for playback.

FIG. 6 is a flowchart of an example process 600 for generating text fromrecorded audio and posting a message that includes the generated text toan online social messaging platform. For convenience, the process 600will be described as being performed by a user device programmedappropriately in accordance with this specification. For example, anyone of the user devices 104 a-n of FIG. 1, appropriately programmed, canperform the process 600. In the description below, reference is made toa single user device, although it is understood that more than one userdevice can perform the example process 600 in parallel.

The user device displays 605 a user interface for recording audio. Theuser interface can be, for example, the user interface as illustrated inFIG. 3 and described above, and in general includes an input element forstarting and stopping recording.

The user device records 610 the audio. As described above with referenceto FIG. 2, the user device records audio using a microphone on the userdevice, while displaying a graphic or animation, e.g., an animatedwaveform, indicating that the device is recording.

The user device generates text 615 from detected speech in the audiorecording. The user device can generate the text using any suitablespeech-to-text software technique. In some implementations, the userdevice generates and displays text on the user device while recordingaudio at the same time.

In some implementations, instead of the user device generating the textfrom speech detected in the audio recording, the user device sends theaudio recording to the platform, and the platform processes therecording to generate the text. In some implementations, the user devicemodifies the audio recording according to received input, as describedabove with reference to FIG. 2.

The user device posts 620 an audio message that includes the generatedtext. The user device posts 620 the audio message with the generatedtext automatically, without playback, facilitating rapid disseminationof content to the platform while also provided different formats inwhich users receiving the audio message consume the content. In someimplementations, the user device displays the generated text, thegenerated video, or both, for playback and confirmation before postingthe audio message to the platform.

In some implementations in which the generated text is displayed beforethe audio message is posted, the user device is configured to receiveinput indicating edits to the text. For example, the user device canreceive input to edit text, e.g., because the speech-to-text generationwas not fully accurate, or to provide additional text to include withthe message before being posted. Terms within the generated text canalso be used by the platform for indexing the posted audio message. Theplatform can extract and store terms from the generated text, and laterserve search requests by searching an index of messages posted to theplatform. The index can include the extracted terms for the posted audiomessage.

If the platform imposes a text limit, e.g., character or word limit ontext in messages, then the generated text is edited to conform to thetext limit. The user device can automatically edit the text to includeonly the beginning portion of the generated text, up to the text limit.Alternatively, the user interface displays the generated text, and isfurther configured to receive input, e.g., touch or swipe gestures,indicating which text to remove to meet the text limit. Alternatively,the platform can make an exception for text generated from audio and adifferent text limit could be set for the accompanying text, up to andincluding no limit at all. The accompanying text can be rendered with astyle to match the regularly displayed text, or rendered according to adifferent style, e.g., a font that provides a visual indication to theuser that the text was generated for the associated audio recording.

Embodiments of the subject matter and the actions and operationsdescribed in this specification can be implemented in digital electroniccircuitry, in tangibly-embodied computer software or firmware, incomputer hardware, including the structures disclosed in thisspecification and their structural equivalents, or in combinations ofone or more of them. Embodiments of the subject matter described in thisspecification can be implemented as one or more computer programs, e.g.,one or more modules of computer program instructions, encoded on acomputer program carrier, for execution by, or to control the operationof, data processing apparatus. The carrier may be a tangiblenon-transitory computer storage medium. Alternatively or in addition,the carrier may be an artificially-generated propagated signal, e.g., amachine-generated electrical, optical, or electromagnetic signal, whichis generated to encode information for transmission to suitable receiverapparatus for execution by a data processing apparatus. The computerstorage medium can be or be part of a machine-readable storage device, amachine-readable storage substrate, a random or serial access memorydevice, or a combination of one or more of them. A computer storagemedium is not a propagated signal.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, or multiple processors or computers.Data processing apparatus can include special-purpose logic circuitry,e.g., an FPGA (field programmable gate array), an ASIC(application-specific integrated circuit), or a GPU (graphics processingunit). The apparatus can also include, in addition to hardware, codethat creates an execution environment for computer programs, e.g., codethat constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, or a combination of one or moreof them.

A computer program can be written in any form of programming language,including compiled or interpreted languages, or declarative orprocedural languages; and it can be deployed in any form, including as astand-alone program, e.g., as an app, or as a module, component, engine,subroutine, or other unit suitable for executing in a computingenvironment, which environment may include one or more computersinterconnected by a data communication network in one or more locations.

A computer program may, but need not, correspond to a file in a filesystem. A computer program can be stored in a portion of a file thatholds other programs or data, e.g., one or more scripts stored in amarkup language document, in a single file dedicated to the program inquestion, or in multiple coordinated files, e.g., files that store oneor more modules, sub-programs, or portions of code.

The processes and logic flows described in this specification can beperformed by one or more computers executing one or more computerprograms to perform operations by operating on input data and generatingoutput. The processes and logic flows can also be performed byspecial-purpose logic circuitry, e.g., an FPGA, an ASIC, or a GPU, or bya combination of special-purpose logic circuitry and one or moreprogrammed computers.

Computers suitable for the execution of a computer program can be basedon general or special-purpose microprocessors or both, or any other kindof central processing unit. Generally, a central processing unit willreceive instructions and data from a read-only memory or a random accessmemory or both. The essential elements of a computer are a centralprocessing unit for executing instructions and one or more memorydevices for storing instructions and data. The central processing unitand the memory can be supplemented by, or incorporated in,special-purpose logic circuitry.

Generally, a computer will also include, or be operatively coupled to,one or more mass storage devices, and be configured to receive data fromor transfer data to the mass storage devices. The mass storage devicescan be, for example, magnetic, magneto-optical, or optical disks, orsolid state drives. However, a computer need not have such devices.Moreover, a computer can be embedded in another device, e.g., a mobiletelephone, a personal digital assistant (PDA), a mobile audio or videoplayer, a game console, a Global Positioning System (GPS) receiver, or aportable storage device, e.g., a universal serial bus (USB) flash drive,to name just a few.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on one or morecomputers having, or configured to communicate with, a display device,e.g., a LCD (liquid crystal display) or organic light-emitting diode(OLED) monitor, a virtual-reality (VR) or augmented-reality (AR)display, for displaying information to the user, and an input device bywhich the user can provide input to the computer, e.g., a keyboard and apointing device, e.g., a mouse, a trackball or touchpad. Other kinds ofdevices can be used to provide for interaction with a user as well; forexample, feedback and responses provided to the user can be any form ofsensory feedback, e.g., visual, auditory, speech or tactile; and inputfrom the user can be received in any form, including acoustic, speech,or tactile input, including touch motion or gestures, or kinetic motionor gestures or orientation motion or gestures. In addition, a computercan interact with a user by sending documents to and receiving documentsfrom a device that is used by the user; for example, by sending webpages to a web browser on a user's device in response to requestsreceived from the web browser, or by interacting with an app running ona user device, e.g., a smartphone or electronic tablet. Also, a computercan interact with a user by sending text messages or other forms ofmessage to a personal device, e.g., a smartphone that is running amessaging application, and receiving responsive messages from the userin return.

This specification uses the term “configured to” in connection withsystems, apparatus, and computer program components. That a system ofone or more computers is configured to perform particular operations oractions means that the system has installed on it software, firmware,hardware, or a combination of them that in operation cause the system toperform the operations or actions. That one or more computer programs isconfigured to perform particular operations or actions means that theone or more programs include instructions that, when executed by dataprocessing apparatus, cause the apparatus to perform the operations oractions. That special-purpose logic circuitry is configured to performparticular operations or actions means that the circuitry has electroniclogic that performs the operations or actions.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface, a web browser, or anapp through which a user can interact with an implementation of thesubject matter described in this specification, or any combination ofone or more such back-end, middleware, or front-end components. Thecomponents of the system can be interconnected by any form or medium ofdigital data communication, e.g., a communication network. Examples ofcommunication networks include a local area network (LAN) and a widearea network (WAN), e.g., the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data, e.g., an HTML page, to a userdevice, e.g., for purposes of displaying data to and receiving userinput from a user interacting with the device, which acts as a client.Data generated at the user device, e.g., a result of the userinteraction, can be received at the server from the device.

In addition to the embodiments of the attached claims and theembodiments described above, the following numbered embodiments are alsoinnovative:

Embodiment 1 is a method comprising: receiving, by a first client on afirst user device, a request to record an audio message for a socialmessaging platform, wherein the first client is software running on thefirst user device, wherein the first client is configured to provide auser interface for the platform for a user using the first user devicewho is logged in to a user account on the platform, and wherein therequest is received through the user interface; recording audio througha microphone of the first user device; generating a platform message by(i) generating a video file that includes the recorded audio as an audioportion of the video file and programmatically generated minimal videocontent as a video portion of the video file, and (ii) including thevideo file in the platform message; and posting, by the first client,the platform message to the platform, in response to a post requestreceived by the first client.

Embodiment 2 is the method of embodiment 1, further comprising:receiving, by the first client on the first user device, one or moreedit requests to edit the recorded audio; and in response to the one ormore edit requests and before generating the platform message, editingthe recorded audio.

Embodiment 3 is the method of any one of embodiments 1 or 2, wherein theminimal video content presents an appearance of a moving waveform withina duration of the audio recording, the method comprising: receiving, bythe first client on the first user device, a request to insert a soundeffect at a particular point in the audio recording; receiving, by thefirst client on the first user device, a selection of a visual markercorresponding to a sound effect and a time input indicating when in therecorded audio to include the sound effect, wherein the visual marker isselected from among one or more visual markers displayed on the userinterface; and in response to the selection of the visual marker and thetime input: modifying the recorded audio to play the sound effect at thereceived time input in the recorded audio, and modifying the minimalvideo content to display the visual marker on the moving waveform at aposition corresponding to the time input.

Embodiment 4 is the method of embodiment 3, wherein modifying therecorded audio comprises inserting the sound effect at the received timeinput in the recoded audio.

Embodiment 5 is the method of embodiments 3 or 4, wherein modifying therecording audio comprises overlaying the sound effect over the recordedaudio at the received time input, wherein when the audio recordingincludes both the received time input and the sound effect when playedback at the received time input.

Embodiment 6 is the method of any one of embodiments 1 through 5,wherein the minimal video content is an unchanging image, a monochromeimage, or a loop of a waveform against a static background.

Embodiment 7 is the method of embodiment 6, wherein the waveform is avisual representation of at least a portion of the recorded audio to thefirst user device.

Embodiment 8 is the method of any one of embodiments 1 through 7,wherein the video file is an MPEG-4 file.

Embodiment 9 is the method of any one of embodiments 1 through 8,wherein the method further comprises: receiving, by the first client, aspeech-to-text request; in response to the speech-to-text request:detecting speech in the recorded audio; generating text corresponding tothe detected speech, and displaying the generated text corresponding tothe detected speech, wherein generating the platform message comprisesadding the generated text to the platform message; and using, by thesocial messaging platform, the generated text to index the platformmessage for searching.

Embodiment 10 is a method comprising: receiving, by a client on a userdevice, a platform message comprising a video file, the video filehaving a video portion and an audio portion; and determining, by theclient, that the video portion of the video file comprises minimal videocontent that has been programmatically generated, and in response:obtaining second video content by the client, the second video contentbeing different from the minimal video content, and displaying, on adisplay of the user device, the second video content while playing theaudio portion of the video file of the platform message.

Embodiment 11 is the method of embodiment 10, wherein obtaining thesecond video content comprises generating, by the client, the secondvideo content.

Embodiment 12 is method of any one of embodiments 10 or 11, whereindetermining that the video portion of the video file comprises theminimal video content comprises: determining that a signature of dataspecifying the video portion of the video file matches a predeterminedsignature of data corresponding to programmatically generated minimalvideo content.

Embodiment 13 is a system comprising: one or more computers and one ormore storage devices storing instructions that are operable, whenexecuted by the one or more computers, to cause the one or morecomputers to perform the method of any one of claims 1 to 12.

Embodiment 14 is a computer storage medium encoded with a computerprogram, the program comprising instructions that are operable, whenexecuted by data processing apparatus, to cause the data processingapparatus to perform the method of any one of claims 1 to 12.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of what isbeing claimed, which is defined by the claims themselves, but rather asdescriptions of features that may be specific to particular embodimentsof particular inventions. Certain features that are described in thisspecification in the context of separate embodiments can also beimplemented in combination in a single embodiment. Conversely, variousfeatures that are described in the context of a single embodiment canalso be implemented in multiple embodiments separately or in anysuitable subcombination. Moreover, although features may be describedabove as acting in certain combinations and even initially be claimed assuch, one or more features from a claimed combination can in some casesbe excised from the combination, and the claim may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings and recited inthe claims in a particular order, this should not be understood asrequiring that such operations be performed in the particular ordershown or in sequential order, or that all illustrated operations beperformed, to achieve desirable results. In certain circumstances,multitasking and parallel processing may be advantageous. Moreover, theseparation of various system modules and components in the embodimentsdescribed above should not be understood as requiring such separation inall embodiments, and it should be understood that the described programcomponents and systems can generally be integrated together in a singlesoftware product or packaged into multiple software products.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. For example,the actions recited in the claims can be performed in a different orderand still achieve desirable results. As one example, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In some cases, multitasking and parallel processing may beadvantageous.

What is claimed is:
 1. (canceled)
 2. One or more non-transitorycomputer-readable storage media encoded with instructions that, whenexecuted by a user device, cause the user device to perform operationscomprising: receiving, by a client on a user device, a platform messagecomprising a video file, the video file having a video portion and anaudio portion; and determining, by the client, that the video portion ofthe video file comprises minimal video content that has beenprogrammatically generated, and in response: obtaining second videocontent by the client, the second video content being different from theminimal video content, and displaying, on a display of the user device,the second video content while playing the audio portion of the videofile of the platform message.
 3. The computer-readable storage media ofclaim 2, wherein obtaining the second video content comprisesgenerating, by the client, the second video content.
 4. Thecomputer-readable storage media of claim 2, wherein determining that thevideo portion of the video file comprises the minimal video contentcomprises: determining that a signature of data specifying the videoportion of the video file matches a predetermined signature of datacorresponding to programmatically generated minimal video content. 5.The computer-readable storage media of claim 2, wherein determining thatthe video portion of the video file comprises the minimal video contentcomprises: determining that the video portion includes a flag indicatingthat the video content was programmatically generated.
 6. Thecomputer-readable storage media of claim 2, wherein the programmaticallygenerated video content contributes less, to the file size of the videofile, than the audio portion.
 7. The computer-readable storage media ofclaim 2, wherein the received video file is in an audiovisual format. 8.The computer-readable storage media of claim 2, wherein displaying thesecond video content while playing the audio portion of the video fileof the platform message comprises: rendering, using a codec for anaudiovisual file format, the second video content to substitute theminimal video portion of the video file while playing the audio portionof the video file.
 9. A computer-implemented method comprising:receiving, by a client on a user device, a platform message comprising avideo file, the video file having a video portion and an audio portion;and determining, by the client, that the video portion of the video filecomprises minimal video content that has been programmaticallygenerated, and in response: obtaining second video content by theclient, the second video content being different from the minimal videocontent, and displaying, on a display of the user device, the secondvideo content while playing the audio portion of the video file of theplatform message.
 10. The computer-implemented method of claim 9,wherein obtaining the second video content comprises generating, by theclient, the second video content.
 11. The computer-implemented method ofclaim 9, wherein determining that the video portion of the video filecomprises the minimal video content comprises: determining that asignature of data specifying the video portion of the video file matchesa predetermined signature of data corresponding to programmaticallygenerated minimal video content.
 12. The computer-implemented method ofclaim 9, wherein determining that the video portion of the video filecomprises the minimal video content comprises: determining that thevideo portion includes a flag indicating that the video content wasprogrammatically generated.
 13. The computer-implemented method of claim9, wherein the programmatically generated video content contributesless, to the file size of the video file, than the audio portion. 14.The computer-implemented method of claim 9, wherein the received videofile is in an audiovisual format.
 15. The computer-implemented method ofclaim 9, wherein displaying the second video content while playing theaudio portion of the video file of the platform message comprises:rendering, using a codec for an audiovisual file format, the secondvideo content to substitute the minimal video portion of the video filewhile playing the audio portion of the video file.
 16. A systemcomprising a user device and one or more storage devices on which arestored instructions that are operable, when executed by the user device,to cause the user device to perform operations comprising: receiving, bya client on the user device, a platform message comprising a video file,the video file having a video portion and an audio portion; anddetermining, by the client, that the video portion of the video filecomprises minimal video content that has been programmaticallygenerated, and in response: obtaining second video content by theclient, the second video content being different from the minimal videocontent, and displaying, on a display of the user device, the secondvideo content while playing the audio portion of the video file of theplatform message.
 17. The system of claim 16, wherein obtaining thesecond video content comprises generating, by the client, the secondvideo content.
 18. The system of claim 16, wherein determining that thevideo portion of the video file comprises the minimal video contentcomprises: determining that a signature of data specifying the videoportion of the video file matches a predetermined signature of datacorresponding to programmatically generated minimal video content. 19.The system of claim 16, wherein determining that the video portion ofthe video file comprises the minimal video content comprises:determining that the video portion includes a flag indicating that thevideo content was programmatically generated.
 20. The system of claim16, wherein the programmatically generated video content contributesless, to the file size of the video file, than the audio portion. 21.The system of claim 16, wherein the received video file is in anaudiovisual format.
 22. The system of claim 16, wherein displaying thesecond video content while playing the audio portion of the video fileof the platform message comprises: rendering, using a codec for anaudiovisual file format, the second video content to substitute theminimal video portion of the video file while playing the audio portionof the video file.